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A  crucial  step  for  either  initial  issue  development  or  post  deployment  support  updates  of  military  software  products 
is  integrated  systems  testing.  After  the  development  team  has  conducted  progressive  coding/module  level  tests  and 
appropriate  systems  integration  tests  (using  a  systems  integration  laboratory),  a  proposed  new  version  for  the 
software  product  emerges.  The  critical  issue  becomes  “Is  this  version  ready  for  final  operational  testing  and  for 
release  for  use  by  our  military  users?”.  During  Developmental  Test  and  Evaluation  (DT&E),  each  proposed  new 
version  is  subjected  to  integrated  systems  testing.  The  configuration  item  is  installed  into  representatively 
configured  military  hardware  system  (aircraft,  ships,  etc.).  During  testing,  it  is  operated  in  the  near-operation^ 
environment  (at  sea  or  in  flight,  using  highly  skilled  military  operators  performing  a  carefully  planned  representative 
task).  The  authors  are  team  members  of  Naval  Helicopter  software  development  Integrated  Program  Teams  (IPTs). 
Their  pragmatic  experiences  in  performing  this  value  added  testing  in  the  near-operational  environment  is  useful  for 
consideration  by  all  software  developers  and  testers. 
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1.0 

A 

military  software  products  is  integrated  systems  testing.  After  the  development  team  has 
conducted  progressive  coding/module  level  tests  and  appropriate  systems  integration  tests  (using 
a  systems  integration  laboratory),  a  proposed  new  version  for  the  software  product  emerges. 
The  critical  issue  becomes  "Is  this  version  ready  for  final  operational  testing  and  for  release  for 
use  by  our  military  users?".  During  Developmental  Test  and  Evaluation  (DT&E),  each  proposed 
new  version  is  subjected  to  integrated  systems  testing.  The  configuration  item  is  installed  into 
representatively  configured  military  hardware  system  (aircraft,  ships,  etc.).  During  testing,  it 
is  operated  in  the  near-operational  environment  (at  sea  or  in  flight,  using  highly  skilled  military 
operators  performing  a  carefully  planned  representative  task).  The  authors  are  team  members 
of  Naval  Helicopter  software  development  Integrated  Program  Teams  (IPTs).  Their  pragmatic 
experiences  in  performing  this  value  added  testing  in  the  near-operational  environment  is  useful 
for  consideration  by  all  software  developers  and  testers. 
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INTRODUCTION' 

ucial  step  for  either  initial  issue  developthfeinif  bf  pBS't 'deployment  support  updates  of 


Employees  of  the  Naval  Air  Warfare  Center  (NAWC)  participate  in  a  partnership  of 
organizations  which  work  together  on  the  IPTs  to  provide  high  quality,  technologically  superior 
products  and  support  to  the  customers  of  the  Naval  Air  Systems  Command  (NAVAIRSYSCOM). 
They  are  part  of  the  test  teams  responsible  for  testing  and  supporting  aircraft/related  systems  that 
can  be  operated,  based  or  sustained  at  sea.  The  authors  competency  group  is  the  newly  created 
Tactical  Data  Systems  Branch  (TDSB,  code  4.11.7.1)  in  the  Test  and  Evaluation  Engineering 
Department,  responsible  for  providing  personnel,  processes  and  facilities  to  IPTs.  They  perform 
the  DT&E  of  integrated  avionics  systems,  which  include  new  or  modified  software,  and 
integration  of  software  into  avionics  and  hardware  components. 


In  the  near-operational  environment,  IPTs  conduct  detailed  technical  tests  which  include  _ 
the  use  of  various  shore-based  facilities  with  operationally  equipped  aircraft.  Specialists  in  real¬ 
time  data  acquisition,  range  tracking/control,  acoustics  simulation,  electronic  warfare  simulation, 
and  command  and  control  participate  in  each  airborne  test  program.  They  contribute  the 
necessary  technical  expertise  to  evaluate  system  performance  at  both  the  technical  and  - 
operational  levels.  Aircrews  are  hand  picked  because  of  their  significant  operational  experience 
and  personal  motivation  which  contribute  to  the  realism  built  into  each  test.  Together,  this  - 


‘  Reference  (a)  Naval  Aviation  Systems  Team  1994/95  Strategic  Plan 
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confluence  of  personnel,  equipment  and  technical  insight  provide  the  backdrop  for  the  near- 
operational  environment. 


1.1  SOFTWARE  DEVELOPMENT  PROCESS^ 

As  part  of  the  military  software  development  process,  the  Department  of  Defense  has 
established  a  formalized  series  of  milestones  that  combine  to  control  and  influence  software 
production  process.  Technical  reviews  are  held  for  the  completion  of  major  tasks  before 
committing  further  resources  in  what  may  turn  out  to  be  a  risk-laden  development.  The  purpose 
of  the  software  development  process  is  to  define  requirements  early  in  the  process  and  decide 
how  to  implement  them.  Different  levels  of  testing  are  appropriate  at  distinct  stages  of  the 
process  (code,  module,  simulation/stimulation,  etc.).  DT&E  engineers  follow  the  software 
development  process  and  are  able  to  give  advise  to  software  designers  in  terms  of  mission 
relation.  Participation  by  test  engineers  allows  technical  problems  to  be  caught  early  on  in  the 
development  cycle. 
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Figure  1. 

Software  Development  Process 


^  Reference  (a)  DOD-STD-2167A,  Military  System  Software  Development 
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The  software  development  effort  consists  of  seven  activities,  Figure  1,  which  when 
combined  lead  to  methodically  created  software  developed  and  tested  in  controlled  environments: 
(1)  system  requirements  ansiysis,  (2)  system  design,  (3)  software  requirements  analysis  and 
definition,  (4)  preliminary  design,  (5)  detailed  design,  (6)  coding,  unit  and  module  level  testing, 
and  (7)  DT&E.  At  the  end  of  each  of  these  activities  are  reviews  (System  Requirements 
Review,  System  Design  Review,  Software  Specification  Review,  Preliminary  Design  Review, 
Detailed  Design  Review,  Test  Readiness  Review)  which  are  held  to  determine  the  completion 
of  the  activity  and  the  capability  to  proceed  to  the  next  activity.  IPT  management  personnel 
support  the  development  effort  by  continuously  reviewing  and  updating  the  plan  of  action  and 
milestones,  thus  ensuring  that  the  process  remains  on  track.  Competency  management  also 
supports  the  process  by  providing  the  necessary  resources  (personnel,  aircraft  and  facilities)  to 
complete  each  task  and  qualified  staffing,  for  IPT  support  performance  on  a  periodic  basis.  The 
advantages  of  formalizing  the  development  process  include  lower  development  costs,  shorter 
development  times,  and  fewer  systematic  errors  in  the  finalized  code.  DT&E  project  engineers 
actively  participate  as  IPT  team  members  throughout  the  software  development  process.  Their 
special  role  as  installed  systems  testers  becomes  activated  once  the  proposed  software  product 
reaches  the  DT&E  phase. 


1 . 1 . 1  Software  Developmental  Test  and  Evaluation 

The  goals  of  the  DT&E  are  to  ensure  that  installed  avionics  and  software  systems  meet 
the  software  requirements,  are  mission  ready  and  ready  for  Operational  Test  and  Evaluation 
(OT&E).  DT&E  addresses  the  issues  of  the  appropriateness  of  the  crew-system  interface, 
system  performance  under  near  operational  conditions,  software  reliability  and  software  safety. 
The  ne^  for  control  of  the  test  environment  during  DT&E  is  met  through  the  use  of 
sophisticated  test  facilities  which  include  the  Ship  Ground  Station  (SGS),  the  Helicopter  Mission 
Systems  Support  Center  (HMSSC),  Chesapeake  Test  Range  (CTR)  and  the  Mid- Atlantic  Test 
Range.  They  provide  the  capability  to  execute  full-up  mission  profiles  for  Anti-Subsurface 
Warfare,  Electronic  Support,  Surface  Radar  tracking,  or  weapons  delivery.  The  near-operational 
environment  may  include  multiple  ship/air  platforms  for  certain  test  scenarios,  but  are  typically 
designed  for  simplicity  and  to  minimize  cost. 

Software  DT&E  is  conducted  to  ensure  installed  software  specification  compliance  of  an 
operational  system  in  its  intended  environment.  A  major  portion  of  DT&E  consists  of  planning, 
conducting  tests  and  reporting  test  results  of  aircraft  and  weapon  systems  software.  Testing 
evaluates  the  new  or  modified  systems  performance,  reliability,  interoperability,  safety,  and 
crew-system  interface  throughout  the  range  of  operational  factors.  Test  results  are  analyzed  and 
evaluated  to  determine  requirements’  compliance,  mission  suitability  and  readiness  for  the  user. 
The  overriding  purpose  of  DT&E  is  to  translate  test  data  and  mission  relation  into  information 
for  acquisition  decisions,  future  design  improvements  and/or  deficiency  corrections.  DT&E  is 
conducted  before  OT&E  to  support  the  acquisition  decision  for  initial  operational  capability  or 
supports  the  sequence  of  events  leading  to  lifecycle  updates  of  inservice  software  and  helicopters. 
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1.1.2 


Operational  Evaluation 


In  general,  OT&E  is  the  last  testing  performed  on  the  software  before  it  is  released  to 
the  fleet.  The  purpose  of  OT&E  is  to  evaluate  the  effectiveness  and  suitability  of  the  software 
as  installed  in  the  system.  OT&E  certifies  the  software  for  fleet  release  and  ensures  its 
operational  effectiveness  while  identifying  deficiencies.  It  differs  from  DT&E  in  that  the  aircrew 
and  support  personnel  conducting  the  testing  are  representative  of  the  type  of  personnel  in  the 
fleet.  The  environment  for  OT&E  goes  one  step  beyond  DT&E  in  that  it  mimics  the 
environment  which  will  be  experienced  under  fleet  use  conditions. 


2.0  TEST  METHODOLOGY 

2.1  TEST  AND  EVALUATION  FACTORS 

A  typical  DT&E  evaluation  of  software  consists  of  a  documentation  review  before 
delivery,  then  planned  laboratory,  ground  and  flight  tests  after  software  delivery.  Typical 
software  products  provide  new  functions  and  correct  prior  problems  (previously  document  by 
deficiencies  report).  During  DT&E,  the  following  four  factors  are  considered:  subjects,  criteria, 
procedures  and  controls,  and  setting’. 

(a)  Subjects:  Each  evaluation  is  carried  out  with  personnel  who  use  the  software  in  as 
fleet  operators  will  use  it.  Operators  are  selected  based  on  their  knowledge  of  the  system  being 
tested  and  are  trained  to  recognize  the  performance  differences  of  the  new  software  can  exhibit. 
For  example,  a  test  team  will  include  test  pilots,  highly  skilled  aircrew,  and  software/computer 
professionals. 

(b)  Criteria:  Each  test  effort  consists  of  test  metrics  such  as  system  loading  time,  system 
readiness  criteria  (which  are  based  on  mission  needs),  equipment  maintainability,  and  the  number 
of  software  errors  detected.  For  software,  which  requires  significant  aircrew  dexterity  (e.g.,  like 
intercepting  a  constantly  changing  circular  course),  the  aircrew  evaluate  the  software  against 
standardized  handling  qualities  rating  scales'*. 

(c)  Procedures  and  Controls:  Procedures  for  each  test  evolution  contain  the  step-by-step 
(not  necessarily  keystroke-by-keystroke)  methods  needed  to  produce  the  desired  evaluation 
results.  Each  deficiency  report  (STR  or  Yellow  Sheet),  documented  during  previous  evaluations, 
is  broken  down  into  constituent  steps  and  then  worked  into  a  scenario  which  presents  the 


’  adapted  from  Cognitive  Ergonomics-Understanding,  Learning,  and  Designing 
Human-Computer  Interaction;  1990. 

4  Cooper-Harper  Handling  Rating  Systems,  NASA 
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operator  with  situations  that  he  or  she  is  likely  to  encounter.  These  scenarios  are  written  in 
flight  cards  or  ground  test  procedures.  Since  not  every  problem  possibility  can  be  covered  this 
way,  test  scenarios  focus  on  the  most  likely  to  occur  situations,  plus  high  density  or  stressed 
situations  (large  number  of  targets,  etc). 

(d)  Setting:  To  properly  evaluate  the  system  we  test  it  under  conditions  as  close  as 
possible  to  those  under  which  it  will  be  used  operationally.  The  operational  environment  and 
the  test  environment  must  each  be  separately  analyzed  and  common  elements  must  be  found 
which  fall  within  the  constraints  of  the  project  budget  and  time  allocated.  The  use  of  operational 
equipment,  (ship,  aircraft,  and  mission  data  recording  equipment)  enhances  the  overall 
simulation  of  performing  an  actual  mission  but  can  significantly  increase  the  cost  of  testing  if 
it  must  be  separately  acquired;  the  use  of  common  element  end  items  is  maximized  whenever 
possible.  The  layout  and  organization  of  the  test  facility  are  also  used  to  exploit  the  operational 
characteristics  of  the  equipment. 


2.2  SOFTWARE  DESIGN 

For  software  driven  systems,  the  primary  crew-system  interface  for  data  entry  and  display 
can  be  a  Control  and  Display  Unit  (CDU)  or  Multipurpose  Display  (MPD)  with  keyset  panel. 
The  most  common  functions  to  be  evaluated  for  these  units  are: 

•  Entry  of  data  required  at  initialization  and  data  modification  during  flight, 

•  Upload  and  download  of  data  from  external  devices, 

•  Built-in-Tests  (BIT)  and  check  of  equipment  status, 

•  Entry,  display  and  modification  of  dl  data  pages  and  data  bases, 

•  Data  display  and  control  of  all  communications  and  navigation  radios, 

•  Entry  and  display  of  tactical  and  acoustic  data. 

Two  of  the  most  common  system  designs  include  fixed  function  key  entry  systems  and  menu- 
driven  page  entry  systems.  Each  has  distinctive  advantages  that  affect  the  cost,  weight  and 
crew-system  interface  design. 

2.2.1  Key  Entry 

In  the  point  entry  system  environment,  fixed  function  keys  are  evaluated  for  use  in 
gaining  access  to  functions  critical  to  the  aircrew  operator.  Depression  of  a  key  must  result  in 
a  system  response  to  perform  an  action  or  change  a  mode.  A  selection  of  a  key  can  result  in 
the  action  of  display  of  a  sonobuoy,  fly-to-point,  or  reference  mark  on  the  MPD.  A  mode 
change  can  be  caus^  by  a  single  key  entry,  such  as  a  requirement  to  switch  control  from  the 
HELO  controlled  to  a  SHIP  controlled  environment.  Depressions  of  this  key  result  in  the 
aircraft  computer  sending  a  message  to  the  ship  to  take  control  of  certain  airborne  subsystem 
functions.  The  point  entry  system  is  developed  so  that  when  a  key  has  been  selected  it  is  backlit 
active  (green)  and  all  corresponding  selectable  keys  are  backlit  available  (white).  All  keys  that 
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are  not  available  are  not  illuminated.  This  system  allows  the  aircrew  to  make  quick  responses 
to  data  entry  tableaus.  For  mission  systems  involving  a  relatively  flat  menu  structure,  the  point 
entry  system  often  is  cost  effective  and  has  minimal  impact  on  operator  training. 

2.2.2  Menu  Driven  System’^ 

Most  menu  systems  are  evaluated  as  hierarchical  trees,  verifying  each  node  (menu  panel) 
in  the  hierarchy  can  be  reached  only  from  a  single  superordinate  node  that  lies  directly  above 
it  in  the  hierarchy.  In  military  aircraft  systems,  options  may  include  a  choice  of  weapons, 
launch  modes  to  employ,  creation  of  flight  plans,  and  communication  plans.  In  most  of  these 
systems  the  information  is  organized  in  data  pages.  Pages  are  linked  lists,  like  "trees"  where 
the  "root"  of  the  tree  is  reached  by  a  single  keyboard  switch  (NAV,  COM,  IFF,  etc.). 
Subsequent  pages  inside  a  list  or  tree  are  accessed  using  a  line  select  switch  or  a  toggle  switch. 
Certain  functions  and  text  are  dynamic  and  are  removed  from  the  display  in  situations  where  the 
function/text  does  not  apply.  The  options  for  each  panel  will  determine  the  software  menu 
pathways,  the  sequence  of  selections  that  will  be  required  to  get  from  one  menu  panel  to  next. 
Menu-driven  systems  must  handle  a  complex  menu  structure  but  are  limited  by  the  amount  of 
the  workload  imposed  on  the  aircrew.  The  operator  workload  on  various  phases  of  a  typical 
mission  is  a  key  evaluation  criteria  (often  qualitatively  assessed). 


2.3  TESTING  APPROACHES 
2.3.1  System  Performance 

System  performance  is  measured  by  executing  the  test  plan  and  procedures  designed  by 
the  project  engineers.  Software  test  procedures  are  designed  to  execute  specific  software 
functions  within  the  context  of  operational  scenarios.  While  not  every  procedure  is  based  on  an 
operational  scenario,  each  proc^ure  will  specify  a  series  of  steps  executed  with  operational 
equipment.  For  example,  to  evaluate  the  performance  of  a  tracking  algorithm  generated  by  a 
RADAR  system,  a  scenario  would  be  devised  which  would  allow  external  control  of  target 
parameters  (such  as  course,  speed,  altitude  and  heading  and  require  the  operator  to  select  a 
tracking  mode,  filter  integration  time,  display  configuration)  and  interact  with  the  tactical  display 
system.  Variations  in  the  aircraft  flight  parameters  can  be  accounted  for  through  on-board  data 
extraction  or  RADAR  range  tracking  in  circumstances  where  additional  costs  are  warranted. 


5  Reference  (a)  Paap,  K.,  and  Roske-Hofstrand,  R.  (1986).  The  Optimal  Number  of 
Menu  Options  per  Panel.  The  Journal  of  Human  Factors  Society,  Vol.  28  No.4  August 
1986. 
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Measures  of  system  performance  include  subjective  evaluations  by  the  aircrew  and/or 
system  operators.  Technical  metrics  data  includes  measurement  times  for  system  loading, 
initialization,  data  bus  loading,  data  precision  characteristics,  subsystem  response  times  and  data 
latency,  operator  workload,  and  other  parameters  required  to  evaluate  the  system  performance 
with  the  new  software  configuration.  Aircraft  are  required  to  be  mission  ready  within  a  given 
time  period.  System  performance  testing  verifies  the  capability  of  the  aircraft  to  load  and  be 
ready  for  use  in  this  time. 

Software  system  tests  are  conducted  using  prearranged  flight  profiles  designed  to  provide 
technical  data  on  one  or  more  parameters  associate  with  a  particular  function  or  set  of  functions 
in  scenarios  which  are  as  close  to  being  mission-related  as  possible.  For  example,  Anti- 
Subsurface  Warfare  testing  comprises  the  use  of  target  tracking  functions  used  against  real  or 
simulated  targets.  Electronic  Warfare  suites  are  usually  baselined  by  using  real  aircraft  in 
simulated  environments,  but  ultimately  get  tested  in  flight  on  a  test  range  against  actual 
electronic  systems.  The  use  of  flight  time  is  maximized  by  extensive  use  of  aircraft  ground  test 
facilities.  The  technical  parameters  are  controlled  so  that  post-mission  data  analysis  will  yield 
performance  parameters  as  they  are  called  out  in  the  project  test  plan. 

2.3.2  Crew-System  Interface® 

The  introduction  of  on-board  computers  has  changed  the  role  of  the  test  pilot  and  project 
engineer  in  terms  of  crew-system  interface.  A  goal  in  evaluating  the  crew-system  interface  is 
to  recognize  the  underlying  cognitive  structure  of  a  computer  task  and  determine  its  effects  on 
the  operator’s  performance  in  a  divided-attention  context.  When  designing  equipment,  computer 
software,  and  procedures  for  use  in  military  environments,  it  is  essential  to  understand  the 
relationship  between  design  features  and  the  capability  of  the  operator  to  perform  all  timesharing 
duties  safely  and  efficiently.  Compatibility  of  control/display  interfaces  and  procedures  with 
human  information  processing  capability,  decision  making  effectiveness,  and  limits  of  short-term 
and  long  term  memory  and  computational  skills  are  of  paramount  importance.  The  optimal 
matching  of  the  software  (i.e.,  the  "thought  processes"  of  the  aircraft  systems)  to  the  thought 
processes  of  the  pilot,  has  become  perhaps  the  major  human  factor  challenge  in  the  design  of 
civil/military  aircraft  computers.  If  the  pilot  does  not  understand  the  frame  of  reference  and  the 
logical  processes  underlying  an  aircraft’s  automated  systems,  a  communication  breakdown 
between  pilot  and  aircraft  may  be  the  result.  Consequently,  the  decision-making  capability  of 
the  pilot  or  crew  would  be  severely  degraded.  The  aircraft  computers  have  to  be  user-friendly 
and  must  allow  a  meaningful  dialogue  with  the  aircrew  to  facilitate  decision  making.  The  goal 
is  to  create  user-friendly  computer  systems,  not  a  computer-friendly  aircrew. 


6  References  (a)  Lee,  R.B.  (1989)  Communications  and  Decision  Making  in  the  Glass 

Cockpit,  Human  Factors  7  Aviation  Medicine,  Vol.  36  No.6 


Nov/Dec  1989 

(b)  Salvendy  G.  (1987).  Handbook  of  Human  Factors.  John  Wiley  &  Sons, 


Inc. 
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As  part  of  the  evaluation  of  software  display  presentation,  the  following  components  are 
analyzed:  the  expected  number  of  alternatives  that  a  user  examines  on  a  page,  the  time  required 
to  read  or  process  an  alternative,  key-press  time,  the  time  required  to  register  a  choice,  and  the 
time  required  to  accomplish  the  ultimate  task.  Test  procedures  are  developed  to  exercise  specific 
mission  tasks  and  evaluate  the  crew-system  interface.  Flight  crews  provide  subjective 
evaluations  of  menu  structures,  data  entry  capability,  function  usability,  and  provide  considerable 
insight  on  system  setups  during  flight  testing. 

Some  variables  that  have  been  found  to  affect  the  relative  merits  of  the  crew-system 
interface  in  relation  to  entry  and  selection  in  a  computer-driven  system  include  familiarity  with 
the  software,  the  difficulty  users  have  in  spelling  items  in  the  program  (when  alphanumeric 
entries  are  required),  the  total  size  of  the  data  base,  whether  entry  methods  are  aided,  and  the 
number  of  menu  levels  in  the  design. 


2.3.3  Software  Reliability 

Software  reliability  is  an  important  aspect  of  the  software  development  testing  process. 
This  process  begins  with  the  planning  phase  and  continues  through  the  software  lifecycle.  During 
the  development  phase,  software  reliability  is  maintained  through  oversight  of  the  developer’s 
activities  and  independent  analysis  of  it’s  generated  data  items  (i.e.,  software  metrics,  software 
requirements  analyses,  program  performance  specifications,  etc.). 

While  keeping  in  mind  the  overall  system  design  and  its  mission,  test  engineers  review 
and  evaluate  software  requirements,  design  specifications,  and  the  software  development  plan 
to  ensure  that  the  developer  is  following  the  agreed  upon  software  development,  is  addressing 
all  of  the  system  requirements,  and  is  meeting  project  milestones.  At  each  milestone,  the 
contractor  must  present  the  status  of  each  contract  deliverable  for  the  software  and  review  the 
progress  to  date  for  each  task.  All  of  this  assures  the  government  that,  when  the  development 
process  is  finished,  the  final  product  will  be  supportable  and  maintainable. 

During  the  development  phase,  NAWCAD  maintains  a  trouble  report  tracking  system  that 
allows  for  following  contractor  progress  and  monitoring  contractor  activities.  Test  engineers 
constantly  review  the  latest  developments  in  the  project  by  visiting  the  developer’s  site  and 
attending  program  reviews.  When  deliverables  are  due,  test  engineers  are  responsible  for 
reviewing  documentation,  submitting  comments  where  appropriate,  and  attending  document 
inspections  as  government  witnesses. 

Software  reliability  includes  software  loading  verification  testing,  regression  testing  and 
validation  testing.  Software  loading  verification  is  the  capability  to  load,  reload  during  flight, 
and  successfully  load  the  software  within  time  limits  or  mission  requirements.  Regression 
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testing  is  "selective  retesting  of  a  system  or  component  to  verify  that  modifications  have  not 
caused  unintended  effects  and  that  the  system  or  component  still  compiles  with  its  specified 
requirements’. "  Validation  testing  addresses  corrections  or  enhancements  to  the  new  software 
version  to  ensure  that  the  new  requirements  imposed  on  the  system  are  actually  implemented. 
So  unlike  the  process  of  verification,  which  seeks  to  determine  if  all  of  the  software 
requirements  are  met  by  the  proposed  code  changes,  validation  is  the  process  of  ensuring  that 
the  code  meets  all  the  system  software  requirements. 

Software  stress  testing  is  another  measure  of  software  reliability  used  to  predict  the 
performance  of  a  software  system  under  dynamic  conditions.  Software  stress  test  procedures 
include  using  maximally  loaded  libraries,  inventory  tables,  and  symbology  displays  besides 
creating  equipment  faults  or  processing  high  cyclic  rate  tasks.  It  consists  of  procedures  that 
maximize  the  cyclic  processing  requirements  of  the  software  and  involves  running  as  many 
functions  as  possible  at  the  same  time  to  test  the  limits  of  the  computer  system.  Additionally, 
the  software  is  run  for  two  or  more  mission  cycles  without  reloading.  This  part  of  stress  testing 
will  determine  if  the  system/software  is  capable  of  performing  for  extended  periods  without 
failure. 

2.3.4  Software  Safety 

All  approaches  to  software  testing  are  critical  and  can  impact  the  safety  of  the  software. 
If  the  software  does  not  perform  as  predicted,  this  can  cause  confusion  and  mission  degradation, 
thus  impacting  mission  accomplishment  and  ultimately  the  safety  of  the  aircrew.  Safety  must 
address  how  negative  system  performance  would  impact  mission  accomplishment  and  the  safety 
of  the  aircrew  and  surrounding  friendly  forces. 

As  part  of  the  development  process,  the  contractor  is  required  to  perform  a  software 
safety  analysis.  Usually,  the  safety  analysis  is  performed  with  a  failure  analysis  that  points  out 
single-point  failures  that  can  occur  during  system  operation.  The  safety  analysis  reviews  software 
requirements,  software  design,  operating  procedures  and  identifies  potential  hazards  due  to  the 
software.  For  each  hazard  sited,  a  risk  assessment  is  performed  and  a  mitigation  plan  is 
formulated.  During  DT&E,  the  safety  analysis  is  used  to  formulate  test  procedures  that  provide 
a  safety  margin  to  the  aircrew  in  case  of  failure. 

The  crew-system  interface  addresses  how  safety  information  is  presented  to  the  operator. 
The  alert  presentation  on  a  display  is  very  important  for  aircrew  safety.  For  example,  if  the 
aircrew  is  not  properly  notified  by  display  or  aural  indication  of  a  change  in  status  of  the  aircraft 
(e.g.,  low  altitude  alert,  inertial  navigation  set  failure),  the  results  can  be  catastrophic.  In  such 
cases,  multiple  paths  are  added  to  the  software  to  present  the  warning  indication.  As  a  result, 
the  indication  may  appear  visually  and  trip  an  aural  alarm,  so  if  the  pilot  does  not  see  the  first 
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indicator  he/she  will  hear  the  second  one.  In  cases  where  the  crew  survivability  is  at  stake, 
warning  indicators  are  hardwired  into  the  appropriate  systems. 

Frequently  the  software  safety  aspects  are  not  always  clearly  identified,  as  when  an  alert 
or  mode  change  is  displayed  to  an  operator.  At  times  a  complex  evaluation  of  many  inputs  from 
all  testing  approaches  is  required  to  determine  the  level  of  criticality  of  the  software  hazard  to 
mission  accomplishment  and  suitability  of  the  software  for  fleet  use. 


3.0  RESULTS 

Data  reduction  for  software  flight  testing  involves  deciding  the  input-output  characteristics 
of  the  underlying  subsystem  and  measuring  quantities  that  explain  software  performance.  Test 
data  is  either  electronic  or  handwritten  in  the  form  of  data  cards.  Within  the  HMSSC  at  Rotary 
Wing,  the  LAMPS  MK  HI  program  has  developed  the  ALADIN  (Automated  LAMPS  Data 
Integrated  Network)  system  to  perform  the  task  of  data  reduction  and  conversion  into 
engineering  reports.  Other  systems  in  use  include  the  HV  data  logger,  Loral  SBA-100  Bus 
analyzer,  and  the  MARS  2000  Tape  Instrumentation  System.  For  example,  the  LAMPS  data 
analysis  routines  were  developed  in-house  at  NAWCAD,  but  software  such  as  BBN  Probe  and 
PV  Wave  are  available  as  well.  The  cognizant  engineer  must  decide  which  equipment  will  best 
capture  the  data  he  or  she  wants  and  which  software  package  will  display  that  data  most 
expeditiously. 


3.1  DATA  COLLECTION 

3.1.1  Manual  Data  Collection 

Even  in  today’s  electronic  environment,  the  manual  collection  of  data  is  still  required 
and  is  one  of  the  best  ways  to  ensure  that  some  data  is  collected,  especially  if  the  electronic 
means  fail.  During  ground  events,  project  engineers  manually  record  information  on  flight  data 
sheets  and  constantly  observe  the  system  for  actions  and/or  events  that  do  not  appear  correct. 
Manual  inspection  of  displays  and  recording  of  results  are  at  the  forefront  in  determining  if 
anomalies  exist  in  the  system. 

Pilots  and  aircrew  are  a  valuable  asset  for  data  collection.  Each  flight  is  thoroughly 
debriefed  and  the  aircrew  is  queried  regarding  the  suitability  and  ease  of  use  of  the  software  and 
its  functions.  After  the  flight  event,  the  aircrew  provides  a  written  report  to  the  test  engineer. 
Debriefings  and  flight  report  dailies  are  means  by  which  operators  provide  a  fleet  operator’s 
perspective  of  the  software  and  system. 

The  aircrew  manually  records  information  during  the  flight.  Before  the  flight,  the  project 
engineer  briefs  the  aircrew  on  the  purpose  of  flight,  procedure(s),  and  what  data  is  to  be 
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recorded.  The  aircrew  records  this  information  on  kneeboard  data  cards,  and  other  information 
that  they  deem  to  be  of  value.  The  project  engineer  must  be  diligent  in  creating  the  flight 
cards/data  cards  to  include  not  only  direct  data  that  will  be  recorded  but  also  any  relative  data 
that  may  explain  unexpected  results.  Flight  cards  should  contain  contingency  steps  to  be  taken 
when  unexpected  results  do  occur. 

3.1.2  Test  Instrumentation 

Test  instrumentation  is  equipment  installed  on  the  aircraft  that  is  not  part  of  the  aircraft 
operational  configuration.  The  express  purpose  of  the  test  instrumentation  is  to  retrieve  data  for 
use  in  post-test  analyses.  NAWCAD  capabilities  also  include  real-time  telemetry  for  tests 
involving  flight  critical  maneuvers.  Approval  for  instrumentation  is  granted  locally  but  for  those 
tests  involving  flight  controls  or  aerodynamics,  approval  authority  rests  with  TEAM  4.3. 

3. 1.2.1  Data  Bus  Instrumentation 

Most  Navy  aircraft  use  an  MIL-STD-1553  data  bus  to  control  aircraft  sensors  and  read 
data  from  the  sensors  for  processing.  Data  bus  instrumentation  is  used  to  trap  communication 
between  the  mission  computer  and  remote  terminals  (RT)  or  obtain  data  directly  from  a  terminal. 
This  instrumentation  can  be  set  up  to  either  monitor  the  bus  during  flight  or  simulate  the 
responses  of  one  or  more  terminals.  Once  a  problem  is  cited,  the  bus  analyzer  can  be  used  to 
troubleshoot  or  to  simulate  the  problem. 

Ruggedized  computer  systems  are  used  to  retrieve  data  through  special  test  ports.  The 
ruggedized  systems  allow  for  the  installation  of  flexible  test  instrumentation  systems  on  the  test 
aircraft  without  requiring  extensive  vibration  analysis  of  that  equipment.  Specially  designed  and 
pre-approved  pallets  are  used  to  secure  the  instrumentation  in  place  in  the  aircraft  and  tie  the 
power  systems  directly  into  the  avionics  system  electrical  bus.  This  means  that  once  a  project 
is  approved  locally,  installation  and  checkout  of  the  on-board  systems  can  begin  immediately. 


3.1.3  Video/Audio  Recorders 

Video  and  audio  recorders  are  installed  on  the  aircraft  to  capture  real-time  information 
about  what  is  occurring  on  the  aircraft.  The  audio  recorder  will  capture  flight  crew 
conversations,  air  traffic  control  instructions  and  test  engineer  instructions.  Video  recorders  can 
be  focused  on  the  displays  and  keysets  to  allow  reconstruction  of  the  mission  for  software 
analysis.  Together,  this  instrumentation  provides  a  full  mission  playback  capability  within  the 
HMSSC. 

3.1.4  On-line  Data  Extraction  Systems 

On  line  data  extraction  systems  are  equipment  or  software  that  is  part  of  the  fleet  aircraft 
configuration  or  part  of  kit  installed  into  a  fleet  aircraft.  These  systems  allow  test  engineers  to 
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evaluate  the  performance  of  the  aircrew,  the  aircraft  subsystems,  or  the  aircraft  computer  system 
under  more  realistic  scenarios  because  recording  goes  on  continuously  without  operator 
intervention.  This  data  can  be  used  to  evaluate  aircrew  or  software  performance  when  the 
aircraft  cannot  be  flight  tested  (such  as  when  fleet  aircraft  are  used  to  evaluate  new  products 
or  techniques). 

3. 1.4.1  Mission  Recorders 

The  Mission  Tape  Recorder  Unit  provides  a  permanent  record  of  mission  information. 
Extraction  and  recording  of  information  include  tactical  data,  navigation  parameters,  system 
status  parameters,  tactical  situation  parameters,  acoustic  data,  voice  communication  and  mission 
time.  These  tapes  can  be  played  back  in  the  HMSSC  to  reconstruct  a  flight. 

3. 1.4.2  Mission  software  data  extraction 

On  some  aircraft  the  capability  to  extract  data  is  built  into  the  mission  software.  The  SH- 
60B  mission  software  data  extraction  program  automatically  logs  parameters  computed  by  each 
subsystem  functional  area  and  records  the  data  in  fixed  length  records  on  the  aircraft’s  magnetic 
tape  system.  Included  within  this  parameter  set  are  the  host  computer  performance  measures 
such  as  I/O  times,  number  of  interrupts  issued,  module  processing  time,  I/O  faults  recorded  and 
I/O  utilization  parameters.  The  data  extraction  function  also  records  data  that  can  commonly 
be  found  on  the  aircraft  1553  bus  such  as  mark-on-top  positions,  navigation  parameters,  radio 
tuning  commands,  keyset  depressions,  and  others. 


3.1.5  Laboratories 

The  use  of  laboratories  for  simulation  and  scenario  control  is  a  vital  part  of  testing. 
Laboratories  provide  support  for  engineering  development  testing.  Independent  Verification  and 
Validation,  DT&E,  and  provide  a  variety  of  capabilities: 

•  To  control  or  create  part  of  the  environment, 

•  To  collect  data  during  testing, 

•  To  analyze  data  collected  during  testing, 

•  To  recreate  flight  tests, 

•  To  troubleshoot  problems, 

•  To  simulate  testing  before  flight  testing, 

•  To  review  test  procedures  or  documentation. 

Some  laboratories  provided  for  use  by  the  TDSB  to  the  IPTs  are  described  below. 

3. 1.5.1  Ship  Ground  Station 
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The  SGS  is  used  to  support  DT&E  for  airborne  components,  shipboard  electronics,  and 
the  entire  LAMPS  MK  III  integrated  ship/air  system.  The  SGS  is  capable  of  establishing  a  data 
link  between  test  aircraft  or  fleet  configured  aircraft  and  the  AEGIS  Combat  Systems  (ACSC), 
Wallops  Island,  VA,  for  simulation  in  an  AEGIS  system  combat  environment.  The  SGS  can 
also  be  linked  via  the  ACSC  to  an  SH-60B  helicopter  flying  off  shore  over  the  Atlantic.  SGS 
includes  a  production  configured  NAVSEA  shipboard  TACNAV  bursted  data  link  capability.  To 
simulate  underwater  targets,  the  SGS  is  equipped  with  a  production  On-Board  Trainer  (OBT) 
which  can  provide  the  aircraft  with  a  complete  ASW  scenario. 


3. 1.5. 2  Chesapeake  Test  Range 

CTR  provides  telemetry,  tracking  data  and  range  control  all  in  a  centralized  workstation 
allowing  analysis  and  display  of  flight  measurements  in  real-time.  Other  capabilities  include 
electronic  warfare  threat  simulation  and  long  range  tracking  providing  meter-level  Time-Space- 
Position  Information.  CTR  develops  specialized  instrumentation  packages  and  methods  to 
increase  airborne  data  accuracy  and  throughput. 

3. 1.5.3  Helicopter  Mission  System  Support  Center 

The  HMSSC  located  at  the  Rotary  Wing  Aircraft  Test  Squadron  hangar  supports  all 
rotary  wing  flight  tests  with  computer  hardware  and  software  necessary  to  perform  a  complete 
mission  analysis.  This  laboratory  houses  hot  benches  for  several  helicopter  types,  provides  all 
the  necessary  ground  and  flight  instrumentation  and  has  a  complete  aircraft  communication  suite 
including  the  LAMPS  datalink.  It  also  houses  several  computer  systems  with  the  latest  data 
storage  and  retrieval  technology  used  for  real-time  or  post  flight  data  recovery  and  analysis.  The 
HMSSC  maintains  and  provides  the  most  modern  test  instrumentation  and  measurement 
equipment. 

The  Automated  LAMPS  Data  Integrated  Network  (ALADIN)  is  located  within  the 
Helicopter  Mission  System  Support  Center,  to  aid  in  providing  engineering  and  technical  support 
to  test  and  evaluate  the  LAMPS  MK  III  helicopter.  ALADIN  performs  the  task  of  converting 
magnetic  tape-recorded  data  into  engineering  units  that  can  be  tabularized,  plotted,  or 
summarized  in  statistical  form.  The  aircraft  data  extraction  program  automatically  logs 
parameters  required  by  each  subsystem  functional  area  to  analyze  subsystem  performance.  The 
ALADIN  system  can  retrieve  any  of  the  77  different  stored  messages  and  plot  and/or  tabularize 
the  data. 

The  HMSSC  house  two  avionics  test  benches.  The  AN/ASN-123  TACNAV  test  bench 
and  the  SH-2G  System  Test  Bench  (STB)  are  used  to  test  tactical  data  transfers  with  other  ASN- 
150  and  ASN-123  platforms.  These  test  benches  are  utilized  to  troubleshoot  problems,  explain 
new  software  updates  to  the  aircrew  and  to  replay  tests.  They  are  fully  interconnectable  via  1553 
data  busses  and  can  simulate  a  variety  of  conditions  found  on  the  aircraft. 


13 


3.2  TEST  RESULTS  REPORTS 


Software  problems  are  reported  to  the  sponsor  and  the  Software  Support  Activity  (SSA) 
via  Software  Trouble  Reports  (STRs)  or  Software  Change  Requests  (SCRs).  An  STR  is  used  for 
reporting  an  identified  software  discrepancy  or  to  request  a  maintenance  change  or  analysis  of 
a  problem  of  unknown  origin,  (i.e.,  software  is  not  operating  according  to  approved 
specifications).  An  SCR  is  a  request  for  a  minor  enhancement  (i.e.,  the  need  is  not  contained 
in  an  approved  specification,  but  requested  to  be  implemented  in  the  software).  All  system 
problems  discovered  during  DT&E  (whether  hardware  or  software)  are  reported  in  Board  of 
Inspection  and  Survey  Yellow  Sheet  Deficiencies  Reports. 

Test  results  are  reported  via  officially  approved  test  reports.  NAWCAD  has  four  types 
of  reports  used  to  communicate  test  results  between  testers,  developers  and  program  sponsors. 
The  four  types  of  reports  are: 

•  Quarterly  Status  Report 

•  Message  Report 

•  DT/OT  Transition  Report 

•  Report  of  Test  Results 

All  reports  presents  technical  information  in  different  formats  depending  on  what  data  and  how 
much  data  the  sponsor  requires.  Decisions  on  which  type  of  reports  to  publish  are  established 
with  the  project  sponsor  during  the  project  preliminary  phase  before  DT&E. 


4.0  CONCLUSION 

DT&E  engineers  from  a  competency-aligned  organization  such  as  TDSB,  use  test  plans, 
modem  laboratories,  test  instmmentation  and  data  collection  facilities  to  evaluate  software  used 
on  fleet  configured  aircraft  operated  by  Navy  aircrew.  The  successful  development  and  testing 
of  software  to  operate  Navy  helicopters  are  necessary  for  the  timely  delivery  of  software-based 
technology  to  the  fleet. 

The  TDSB  has  been  providing  the  personnel,  processes  and  facilities  for  IPTs  that  are 
responsible  for  supporting  the  software  life  cycle  on  a  variety  of  helicopters.  Advanced  software 
systems  of  today  require  detailed  management  throughout  the  software  life  cycle.  Test  planning 
along  with  the  use  of  scripted  scenarios  enables  DT&E  engineers  to  evaluate  complex  software. 

Testing  in  the  near-operational  environment  is  a  proven  and  effective  way  of  detecting 
functional  anomalies  of  new  software  products,  allowing  corrections  before  the  fleet  issue 
release.  This  has  proven  to  be  a  key  vdue  added  step  for  military  software  IPTs. 
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